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Method and A p nar;»tu9 for deriving 3D Repres entation 

The present invention relates to a method and apparatus for deriving a representation 
of the three-dimensional (3D) shape of an object from an image (referred to herein 

^ as an object image) of the projection of structured optical radiation onto the object 
surface. The term "structured optical radiation" is a generalisation of the term 
"structured light" and is intended to cover not only structured light but also 
structured electromagnetic radiation of other wavelengths which obeys the laws of 

10 optics. 

In principle the 3D shape of part of an object surface can be obtained by projecting 
structured light, eg a grid pattern onto a surface of the object, acquiring an image of 
the illuminated region of the object surface and identifying the elements of the 
structured light (eg the crossed lines of the grid pattern) which correspond to the 
respective features (eg crossed lines) of the image, assuming that the spatial 
distribution of the structured light is known. 



20 



One such arrangement is shown by Hu & Stockman in "3-D Surface Solution Using 
Structured Light and Constraint Propagation" in IEEE Trans RAMI Vol 2 No 4 pp 
390 - 402. 1989. who discuss the advantages of the technique over stereoscopic 
imaging techniques in which two cameras acquire overlapping images of a common 
surface region of an object and the 3D shape is reconstructed from a correlation of 
25 pairs of features of the two images which correspond to the same physical surface 
feature. In particular the "correspondence problem" of correlating features in the two 
images obtained in stereoscopic imaging reduces to a much simpler "line labelling 
problem" when stripes are projected onto the object surface. Various geometric 
constraints are disclosed which simpKfy this problem further. The projection of 
^° both grids and grey scale patterns is disclosed. 

In the arrangement shown in WO 93/03579 two projectors each project a set of light 
planes diverging from a line origin, the sets of light planes being mutually 
orthogonal and being directed onto a common region of the object surface to form a 
grid pattern. The line origins intersect at a point between the projectors above the 
object; this point is effectively the origin (optical centre) of a virtual projector. 

A camera is arranged to acquire an image of the grid pattern as distorted by the 
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object surface and is so oriented with respect to the projector arrangement that the 
plane defined by a) the baseline joining the optical centres of the virtual projector 
and b) the optical axis of the camera is nearly parallel to one set of light planes. 
This ensures that the stripes in the image corresponding to this set of light planes 
cross only a small number of stripes derived from the other set of light planes and 
that the number of possible pairs of light planes corresponding to a given 
intersection of stripes in the acquired image is reduced to a minimum. Since each 
intersection can be associated with only one pair of light planes the potential 
10 ambiguity in the image reconstruction is reduced. 

The above problem of ambiguity is addressed in the context of stereoscopic imaging 
by W091/15732 (Gordon) which discloses an arrangement in which a laser scanner 
projects a series of stripes onto the scanned object and left and right cameras detect 
the distorted stripes from the object. 



15 
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It is recognised that a given bright point in the image plane of one camera cannot be 
simply correlated with an illuminated point on the surface of the object because it is 
not known which suipe illuminates that point Accordingly, an arbitrary pixel in the 
stripe in one camera's image plane is selected and a line drawn through the cenUe of 
the camera lens projecting this line out into space. This line is then projected onto 
the image plane of the other camera and the resulting epipolar line in the other 
camera's image plane cuts a number of stripes also imaged on its image plane. Any 
25 one of these points of intersection could in principle correspond to the arbitrary pixel 
mentioned above. The particular point which corresponds is found by projecting all 
the points of intersection back into space and determining which of the resulting 
lines intersects a laser stripe from the laser projector. 

30 US 5,838.428 discloses an arrangement employing successively projected patterns of 
stripes, coded in accordance with a DeBruijn sequence and blurred to enable the 
resolution of the arrangement to be increased by interpolation. The projector 
comprises a mask and a xenon flash tube. The code is such that each element in the 
CCD photodetector used to image the projection of the pattern on the object surface 

35 detects a unique sequence of grey levels which is determined by the 3D position of 
the corresponding surface portion of the object and the structure of the projected 
light. The latter is found by a calibration procedure involving imaging the same 
sequence of projections on a reference plane. 
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It should be noted that there is no correlation of images in the calibration or 
measurement procedure. 

Although the above arrangement avoids the correspondence or line labelling 
problem it has the disadvantage of requiring multiple projections of structured light 
and acquisition of images, increasing the complexity and reducing the speed of the 
arrangement. In particular the shape of a moving or shape-changing object cannot be 
determined. Furthermore the calibration procedure is complex. 

An object of the present invention is to alleviate or overcome at least some 
disadvantages of the prior art. 

In one aspect the invention provides apparatus for generating a 3D representation of 
at least part of an object from an object image of the projection of structured optical 
radiation onto the object surface and from at least one calibration image of the 
projection of the structured optical radiation onto a surface displaced from the object 
surface, the apparatus comprising image processing means arranged to generate 
correspondences between at least one calibration image and the object image and 
optionally a further cahbration image, and reconstruction processing means arranged 
to simulate a first projection of the object image and a second projection linking 
respective correspondences of at least two of the correlated images and to derive said 
3D representation from the mutual intersections of the first and second projections. 

The calibration image can for example be of the projection of the structured optical 
radiation onto a calibration surface or can for example be a further object image 
25 obtained after moving the object relative to the camera used to acquire the initial 
object image and the projector means used to project the structured optical radiation. 

Preferably the first and second projections are fiom a baseline linking an origin of 
the structured optical radiation and a perspective centre associated with the image 
(eg the optical centre of the camera lens used to acquire the image), the 
reconstruction processing means being arranged to derive said baseline from two or 
more pairs of correlated features. This feature is illustrated in Figures 2 and 11 
discussed in detail below. 



20 
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In one embodiment the image processing means is arranged to generate 
correspondences between two or more calibration images and to determine the 
spacing between origins of the first and second projections in dependence upon both 
the correspondences of the two or more calibration images and input or stored metric 
information associated with the calibration images. This feature is illustrated la 
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Figure 11, discussed in detail below. 

In another embodiment the reconstruction processing means is arranged to vary the 
spacing between the origins of the first and second projections in dependence upon a 
scaling variable enterable by a user. In this embodiment a further calibration image 
IS not required- Preferably the apparatus includes means for displaying the 3D 
representation with a relative scaling dependent upon the value of the scaling 
variable. 

Preferably the apparatus includes means for combining two or more 3D 
representations and means for adjusting the relative scaling of the representations to 
enable them to fit each other. 

Preferably the image processing means is arranged to generate said correspondences 
of said images by comparing local radioirletric distributions of said images. 

Suitable algorithms for correlating (generating correspondences between) 
overlapping images are already known - eg Gruen's algorithm (see Gruen. A W 
"Adaptive least squares correlation, a powerful image matching technique"5 Afr J of 
Photogrammetry, remote sensing and Cartography Vol 14 No 3 (1985) and Gruen. 
A W and Baltsavias. E P "High precision image matching for digital terrain model 
generation" Int Arch photo grammetry Vol 25 No 3 (1986) p254) and particularly the 
25 "region-growing" modification thereto which is described in Otto and Chau 
"Region-growing algorithm for matching terrain images" Image and Vision 
Computing Vol 7 No 2 May 1989 p83. all of which are incorporated herein by 
reference. 

Essentially. Gruen's algorithm is an adaptive least squares correlation algorithm in 
which two image patches of typically 15 x 15 to 30 x 30 pixels are correlated (»e 
selected from larger left and right images in such a manner as to give the most 
consistent match between patches) by allowing an affme geometric distortion 
between coordinates in the images (ie stretching or compression in which originally 
parallel lines remain parallel in the transformation) and allowing an additive 
radiometric distortion between the grey levels of the pixels m the image patches, 
generating an over-consUained set of linear equations representing the discrepancies 
between the correlated pixels and finding a least squares solution which miniimses 
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the discrepancies. 

The Gruen algorithm is essentially an iterative algorithm and requires a reasonable 
approximation for the correlation to be fed in before it will converge to the correct 
solution. The Otto and Chau region-growing algorithm begins with an approximate 
match between a point in one image and a point in the other, utilises Gruen's 
algorithm to produce a more accurate match and to generate the geometric and 
radiometric distortion parameters, and uses the distortion parameters to predict 
approximate matches for points in the region of the neighbourhood of the initial 
matching point. The neighbouring points are selected by choosing the adjacent 
points on a grid having a grid spacing of eg 5 or 10 pixels in order to avoid running 
Gruen's algorithm for every pixel. 

Hu et al "Matching Point Features with ordered Geometric. Rigidity and Disparity 
Constraints" IEEE Transactions on Pattern Analysis and Machine Intelligence Vol 
16 No 10, 1994 ppl041-1049 (and references cited therein) discloses further 
methods for correlating features of overlapping images. 

Since the above algorithms were developed for generating correspondences between 
images having poorly defined features (eg aerial photographs) whereas the 
projection of sUuctured light onto an object surface will generate distinct local 
radiometric distributions, the problem of correlation is less critical in the context of 
the present invention. Accordingly the precise correlation algorithm is not critical. 
25 However we have found a number of improvements to the Gruen algorithm, as 
follows: 

i) the additive radiometric shift employed in the algorithm can be dispensed with; 
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ii) if during successive iterations, a candidate matched point moves by more than a 
certain amount (eg 3 pixels) per iteration then it is not a valid matched point and 
should be rejected; 

iii) during the growing of a matched region it is useful to check for sufficient 
contrast at at least three of the four sides of the region in order to ensure that there is 
sufficient dau for a stable convergence - in order to facilitate this it is desirable to 
make the algorithm configurable to enable the parameters (eg required contrast) to 
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be optimised for different environments, and 

iv) in order to quantify the validity of the correspondences between images it has 
been found useful to re-derive the original grid point in the starting image by 
applying the algorithm to the nnatched point in the other image (ie reversing the 
stereo matching process) and measuring the distance between the original grid point 
and the new grid point found in the starting image from the reverse stereo matchmg. 
The smaller the distance the better the correspondence. 

In another aspect the invention provides apparatus for deriving a 3D representation 
of at least part of an object, comprising means for projecting structured optical 
radiation onto the surface of the object, means for acquiring a 2D image of the 
projection of the structured optical radiation, and image processing means arranged 
to derive the 3D representation from the distortion of the structure of the projected 
optical radiation by the object surface, the projected structured optical radiation 
having an irregular radiometric and/or colorimetric distribution. 

In a related aspect the invention provides a method of generating a 3D representation 
of at least part of an object, wherein structured optical radiation is projected onto the 
surface of the object, a 2D image of the projection of the structured optical radiation 
on the surface is acquired, and the 3D represenUtion is derived from the distortion 
of the stmcture of the projected optical radiation by the object surface, the projected 
25 structured optical radiation having an irregular radiometric and/or colorimetric 
distribution. 

In a further aspect the invention provides a method of generating a 3D representation 
of at least part of an object from an object image of the projection of structured 
optical radiation onto the object surface and from at least one calibration image of 
the projection of the structured optical radiation onto a surface displaced from the 
object surface, the method comprising the steps of: 
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i) correlating at least one calibration image with the object image and optionally 
with a further calibration image; 



ii) simulating a first projection of the object image and a second projection of the 
structured optical radiation, and 
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iii) deriving said 3D representation from the mutual intersections of the first and 
second projections. 

^ In a related aspect the invention provides image processing apparatus for deriving a 
3D representation of at least part of an object from a 2D image of the illuminated 
object, the object being illuminated with structured optical radiation projected from 
a location spaced apart from the viewpoint at which the 2D image is acquired, the 

IQ 2D image being correlated with the structured radiation, the apparatus comprismg 
digital processing means arranged to form a 3D reconstruction in which an 
illuminated region of the object in the image extends in a simulated 3D space in 
dependence upon both the correlation and a scaling variable, the scaling variable 
being representative of the separation between the location from which the 

15 structured optical radiation is projected and the viewpoint at which the 2D image is 
acquired. 

Preferred features of the invention are defined in the dependent claims. 

20 

In another aspect the invention provides image processing apparatus for derivmg a 
3D representation of at least part of an object from a 2D image thereof, the object 
being illuminated with structured optical radiation projected from a location spaced 
apart from the viewpoint at which the 2D image is acquired, the 2D image being 
25 correlated with rays of the structured radiation, the apparatus comprising digital 
processing means arranged to form a 3D reconstruction which extends in a simulated 
3D space in dependence upon both the correlation and a scaling variable, the scaling 
variable being representative of the separation between the location from which the 
structured optical radiation is projected and the viewpoint at which the 2D image is 
acquired. 

This aspect of the invention is illustrated in Figure 6. Following a simple calibration 
procedure requiring no knowledge of the position of the camera or the projector 
relative to the object it enables a 3D representation to be generated. This can 

35 

optionally be displayed and scaled or it can be distorted eg for special effects in 
graphics and animation. 



Preferably the apparatus is arranged to derive a further 3D reconstruction from a 
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further 2D image acquired from a different viewpoint relative to the object, the 
combining means being arranged to combine the first-mentioned 3D reconstruction 
and the further 3D reconstruction by manipulations in a simulated 3D space 
involving one or more of rotation and translation, the apparatus further comprising 
^ scaling means arranged to reduce or eliminate any remaining discrepancies between 
the 3D reconstructions by scaling one 3D reconstruction relative to the other along 
at least one axis. 

10 Preferably the apparatus is arranged to display both 3D reconstructions 
simuluneously and to manipulate them in simulated 3D space in response to 
commands entered by a user. 

In one embodiment the apparatus is arranged to perform the manipulations of the 3D 
reconstructions under the control of a computer pointing device. 

In a related aspect the invention provides a method of deriving a 3D representation 
of at least part of an object from a 2D image thereof, comprising the steps of 
illuminating the object with structured projected optical radiation, acquiring a 2D 
image of the illuminated object, correlating the 2D image with rays of the structured 
optical radiation, and digitally processing the 2D image to form a 3D reconstruction 
which extends in a simulated 3D space in dependence upon both the correlation and 
a scaling variable, the scaling variable being representative of the separation 
25 between a location from which the structured optical radiation is projected and the 
viewpoint at which the 2D image is acquired. 

Preferred embodiments of the invention are described below by way of example 
ly with reference to Figures 1 to 11 of the accompanying drawings, wherein: 
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Figure 1 is a schematic representation of apparatus in accordance with all aspects of 
the invention; 

Figure 2 is a sketch perspective ray diagram showing the optical arrangement of the 
apparatus of Figure 1 ; 



Figure 3 shows an object image and a calibration image acquired by the apparatus 
Figures 1 and 2 and the correlation of their features; 
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Figure 4 is a sketch perspective ray diagram showing the use of two reference 
surfaces to locate the camera and projector of the apparatus of Figures 1 and 2 on the 
baseline connecting their respective perspective centres; 

Figure 5 is a flow diagram illustrating the mode of operation of the apparatus of 
Figures 1 and 2 in accordance with the first method aspect of the invention; 

Figure 6 is a ray diagram illustrating both the acquisition of the images utilised by 
the apparatus of Figures 1 and 2 and the regeneration of the object shape by virtual 
projectors of the acquired object image and structured optical radiation; 

Figure 7 is a screenshot illustrating the fitting together of two 3D surface portions of 
the object using the apparatus of Figures 1 and 2; 

Figure 8 is further screenshot showing the scaling of the resulting composite 3D 
surface portion along vertical and horizontal axes; 

Figure 9 is a further screenshot showing the scaling of intersecting 3D surface 
portions to fit each other; 

Figure 10 is a screenshot showing a user interface provided by the apparatus of 
Figures 1 and 2 for manipulating the images and 3D surface portions, and 

Figure 11 is a 3D ray diagram showing the derivation of the projector-camera 
separation from the position of two reference or target surfaces. 

Referring to Figure 1. the apparatus comprises a personal computer 4 (eg a 
Pentium® PC) having conventional CPU. ROM, RAM and a hard drive and a frame 
grabber connection at an input port to a digital camera 1 as well as a video output 
port connected to a screen 5 and conventional input ports connected to a keyboard 
and a mouse 6 or other pointing device. The hard drive is loaded with conventional 
35 operating system such as Windows®95 and software: 
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a) to display images acquired by the camera 1 ; 
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b) to generate correspondences between overlapping regions of images input from 
the camera 1; 

c) to derive from the acquired images the baseline joining the perspective centres of 
5 the camera and projector; 

d) to project images acquired by the camera into a simulated 3D space from virtual 
projectors located on the baseline at a separation selected (with the keyboard or 
pointing device) by the user or determined (thereby creating a partial 3D 
reconstruction); 

e) to scale the partial 3D reconstruction along one or more axes and combine such 
partial 3D reconstructions as illustrated in Figures 7. 8 and 9. and 
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0 to determine the separation of the perspective centres of the camera and projector 
along the baseline from further correlations of object images and calibration images, 
and thence derive an accurate partial 3D reconstruction of the object surface. 

Additionally the software is preferably arranged to correct the images for distortion 
due eg to curvature of field of the camera and projector optics before they are 
processed as described above, either during an initial calibration procedure or as 
part of a ray bundle adjustment process during the processing of the object and 
calibration image(s). Suitable correction and calibration procedures are described by 
Tsai in "An Efficient and Accurate Camera Calibration Technique for 3D Machine 
Vision" Proc IEEE CVPR (1986) pp 364-374 and will not be described further. 

The camera, a Pulnix M-9701 progressive scan digital camera, is shown mounted at 
one end of a support frame F on eg a ball and socket mounting and a slide projector 
PR is shown securely mounted on the other end of the frame. Slide projector PR is 
provided with a speckle pattern slide S and is arranged to project the resulting 
speckle pattern onto the surface of region R of an object 3 which is in the field of 
view of camera CM. 



30 



35 



The intrinsic camera parameters are initially determined by acquiring images of a 
reference plate (not shown) in known positions. The reference plate is planar and has 
an array of printed blobs of uniform and known spacing. The following parameters 
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are determined and are therefore assumed to be known in the subsequent 
description: 

i) focal length of the camera 

ii) distortion parameters of the lenses of the camera and projector 

iii) scale factor 

iv) image coordinates of principal point. 

Additionally the pixel size (determined by the camera manucturer) is assumed to be 
known. 

Optionally, the following extrinsic camera parameters are determined; 

a) camera location 

b) camera orientation. 

Alternatively the camera location and orientation can be taken to define the 
coordinate system relative to which the object surface coordinates are determined. 

Referring now to Figure 2. the camera CM is shown with its perspective centre Oc 
located on a baseline vector V and viewing (initially) a target surface T and 
(subsequently) object 3. The (virtual) origin or perspective centre Op of projector 
PR also lies on baseline vector V and is defined by the optical system of the 
projector comprising field lenses OL and condenser lenses CL. A point light source 
LS such as a filament bulb illuminates sUde S and directs a speckle pattern onto 
(initially) target surface T and (subsequently) the surface of object 3. 

The baseline vector V is found by the following procedure: 

FirsUy an image II (Figure 3) of the region of the surface of object 3 illuminated 
by the projected speckle pattern is acquired and stored in the memory of computer 4 
and an arbitrary group of at least two spaced apart points Ql and Q2 of this region 
are selected as points ql and q2 in the image formed on the photodetector plane PD 
of the camera. The group of points ql and q2 is stored. 

Secondly the object 3 is substituted by target surface T and an image 12 (Figure 3) of 
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the illuminated region of the target surface is acquired by camera CM. The position 
and orienution of the target T relative to the camera are found by acquiring an 
image of the target in the absence of any illumination from the projector, utilising a 
known pattern of blobs BL formed on the periphery of the target. The image 12 is 
stored and and a patch defined by its central point (eg Qn, Figure 3) of the first 
image II is correlated with the corresponding point Pq of the second image 12 by 
selecting a surrounding region R of initially 3x3 pixels and. by comparing local 
radiometric intensity distributions by means of the above-described modified Gruens 
algorithm, searching for the corresponding region R' in image 12 which is allowed to 
be distorted with an affine geometric distortion (eg in the simple case illustrated in 
Figure 3, horizontally elongated). The correlated patch is expanded (up to a 
maximum of 19 x 19 pixels) and the process is repeated. In this manner the 
corresponding point Pn is found. 

This process is repeated to find a large number of pairs of correspondences PQ 
(Figure 3) and in particular to correlate the patches centered on PI. P2 (Figure 2) 
with the points in the group Ql, Q2 (Figure 2). Since the algorithm has a sub-pixel 
resolution, the latter are not necessarily centred on particular pixels. 

In the following geomeuic discussion the correspondences are treated for the sake of 
simplicity as correlated pairs of points but it should be noted that this does not imply 
anything about their topography - in particular it does not imply that they lie at 
corners or edges of the object, for example. 

Referring to Figure 2, the origin (perspective cenUe) of the projector PR will lie at 
the intersection of PlQl and P2Q2. However the 3D locations of these four points 
are not known, only the ray lines from the camera on which they lie. namely plPl, 
qlQl, p2P2 and q2Q2. But the line PlQl will lie in the plane OcPlQl ie plane 
Ocplql which is available form the calibration process and the two images II and 
12 and the line P2Q2 will lie in the plane OcP2Q2 ie plane Ocp2q2 which is 
similarly available from the calibration process and the two images II and 12. These 
planes define a baseline vector V by their intersection, which passes through OC and 
the perspective centre Op of the projector. 

A particularly simple way of finding the baseline vector V is to project plql and 
p2q2 which will meet at a point X in the plan? of photodetector PD. The projection 
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from point X through the perspective centre Ocis the baseline vector V as shown. 

In this manner the baseline vector V can be determined, though not the position of 
the projector origin Op along this baseline. In practice the groups of points P and Q 
will each comprise more than two pairs and hence overdetermine the baseline vector 
V. Accordingly the computer 4 is preferably arranged to derive a bundle of such 
vectors as determined by the sets of points PQ, to eliminate "outliers" ie those 
vectors which deviate by more than a given threshold from the mean and to perform 
a least squares estimate of vector V on the remainder of the vectors, in accordance 
with known statistical methods. 

The derivation of a three-dimensional representation of the object 3 is shown in the 
ray diagram of Figure 6. 

The camera CM and projector PR are shown located on baseline vector V. A first 
virtual projector prl is implemented by the image processing software in computer 4 
and has the same optical characteristics as the camera (as determined in the initial 
calibration procedure). Image II (Figure 3) is projected from this virtual projector in 
a 3D space simulated by the image processing software. 

A second virtual projector pr2 is similarly implemented by the image processing 
software and preferably has the same optical characteristics as the projector PR 
(which is also represented in Figure 6). This virtual projector projects a set of ray 
lines in the simulated 3D space corresponding to the respective physical projector 
rays PQ and the ray lines are each labelled with the respective correlated pixels of 
the image II as found in the image correlation process described with reference to 
Figure 3. It will be appreciated that the image 12 and target T define, and can be 
equated with, a set of rays originating from the perspective centre OP of the 
projector. Since it is known which ray line from the projector PR/prl intersects each 
ray line from its corresponding pixel in image II. the point in 3D space 
corresponding to each intersection can be found, and hence the set of points Qa. Qb. 
Qc... defining the surface. 

In practice many ray lines will not intersect and the best estimate of the 
corresponding 3D surface point will be the mid-point of the perpendicular line 
joining them at their closest approach- Algorithms for this purpoge are known -per se. 
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In the above discussion of Figure 6 it has been assumed that the relative positions of 
the camera CM and and projector PR on baseline vector V (and hence the positions 
of the virtual projectors prl and pr2) are known. In fact these are assumed or entered 
as a scaling variable by the user of the computer 4. 

That the ray lines from the respective virtual projectors will intersect irrespective of 
the spacing between the virtual projectors, assuming that their orientations are 
unchanged, is illustrated in Figure 6 by the alternative virtual projector position pr2' 
corresponding to an assumed real projector position PR'. The resultmg 3D 
reconstruction of the object 3' will be different in size but of the same shape: a 
single scaling factor will be required to interconvert objects 3 and 3'. However it 
may be convenient in practice to provide for different horizontal and vertical scaling 
factors (eg because of different horizontal and vertical magnifications of the camera) 
and in general only one set of scaling factors will be consistent with fitting together 
a set of partial 3D surfaces of the same object acquired from different directions. 

Accordingly the software in computer 4 is arranged to scale such acquired 3D 
representations to enable them to be fitted together to form a self-consistent overall 
3D representation of the object This aspect is described below with reference to 
Figures 7 to 10. 

Before doing so however, an alternative calibration procedure will be described 
with reference to Figure 4. which shows two planar calibration targets Tl and T2 
(having peripheral blobs or discs BL similar to target T of Figure 2) whose 
orientations and positions relative to the camera axis system are known, eg as a 
result of a photogrammetric determination involving separately acquiring images of 
them in the absence of any illumination from the projector, and processing the 
images in a procedure similar to that described above in connection with Figure 2. 
The perspective centres Ocand Op of the camera and projector are also shown. 

In a first stage of the calibration procedure, target Tl is illuminated by the structured 
light from the projector and an image is acquired by the camera CM. Figure 4 
illustrates three points pi. p2 and p3 at which the structured light impinges on target 
Tl. These (and many other points, not shown) will be imaged by the camera CM. 
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In a second stage of the calibration procedure, target Tl is removed and target T2 is 
illuminated by the structured light from the projector. An image is acquired by the 
camera CM. The three points PI. P2 and P3 corresponding to points pi. p2 and p3 
are found by correlating the newly acquired image of the projection of the structured 
radiation on target T2 with the previously acquired image of the corresponding 
projection on Tl by the procedure described above with reference to Figure 3. 

Figure 11 illustrates further the relationship between the positions of two calibration 
targets Tl and T2 and the perspective centre Op of the projector PR (the camera CM 
being assumed fixed on the baseline vector V). A pair of points PI and P2 on target 
Tl form image points pi and p2 respectively on the photodetector array PD of 
camera CM and (in a subsequent step following the removal of target Tl) a pair of 
points P3 and P4 on Urget T2 which are correlated with PI and P2 respectively 
form image points p3 and p4 respectively on photo detector array PD. 

Accordingly in a third stage the pencil of rays formed by corresponding points on 
targets Tl and T2 (eg PI and P3; P2 and P4) is constructed to find the position of the 
perspective centre Op of the projector. In practice the rays will not intersect at a 
point but a best estimate can be found from a least squares algorithm. 

It will be appreciated that other calibration procedures are possible. For example the 
camera could be calibrated by the Tsai method (Roger Y Tsai. IEEE Journal of 
Robotics and Automation RA-3. No 4. August 1987 p 323 - see also references 
cited therein). 

Figures 7 to 10 illustrate the fitting together and manipulation of the partial 3D 
reconstructions on screen. 

In Figure 7 a partial 3D reconstruction Rl and a partial 3D reconstruction R2 are 
fitted together at their corresponding points F and F'. A and A' and result in a 
composite (but still incomplete) 3D reconstruction R3 as shown in Figure 13 having 
an array comprising points E. B and D from Rl and points C and G' in defined 
positions which can be fitted to a further partial 3D reconstruction R4 as shown in 
Figure 8. 

Referring to Figure 8. it is assumed that faces ABC. DGCB. DGEF and ABDEF of 
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R4 are formed but that the region bounded by points A. C . G and F is not. The user 
rotates R3 on screen using eg mouse 6 (Figure 1) to give a routed partial 3D 
representation R3' which is aligned with R4 and which he/she then sees is 
horizontally compressed relative to R4. Accordingly R3 can be expanded 
horizontally or R4 can be compressed horizontally to enable them to be fitted 
together to form a self consistent, complete 3D representation of the object R5. 

As shown in Figure 9, this can be compressed or expanded along any desired axis or 
otherwise distorted under the control of the user to produce a final representation R6 
or R6' having any desired relationship with the actual object 3. 

Figure 10 shows a screen shot of the user interface generated by a program for 
manipulating the 3D reconstructions 30. Buttons BN are provided which can be 
clicked on by the mouse under the control of the user and, when thus selected, enable 
the user to drag portions of the displayed representation so as to zoom, rotate and 
pan the view of the object, as well as to come closer to and move away from the 
object ("nearer" and "further" buttons respectively). As described this far. the 
mterface is similar to the publicly available interface of the COSMO PLAYER web 
browser plug-in produced by Cosmo Software Inc. of California USA. However, in 
accordance with a feature of the present invention, "wheels" Wl and W2 are 
provided which arc rotatable by the mouse and enable the user to adjust the 
separation between the virtual projectors and to vary the distance to the object 
respectively. The latter control is effectively a perspective control. Optionally, 
further buttons or other means (not shown) may be provided to enable distortions 
(such as those described above) to be applied in a graphical fashion, or to enable 
other distortions such as shear distortion to be applied. 



Preferred features of any aspect of the invention can be combined with any other 
aspect of the invention. 
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Claims 



1. Apparatus for generating a 3D represenUtion of at least part of an object from an 
object image of the projection of structured optical radiation onto the object surface 
and from at least one calibration image of the projection of the structured optical 
radiation onto a surface displaced from the object surface, the apparatus comprising 
image processing means arranged to generate correspondences between at least one 
calibration image and the object image and optionally a further calibration image, 
and reconstruction processmg means arranged to simulate a first projection of the 
object image and a second projection linking respective correspondences of at least 
two of the correlated images and to derive said 3D representation from the mutual 
intersections of the first and second projections. 

2. Apparatus as claimed in claim 1 wherein the first and second projections are from 
a baseline linking an origin of the structured optical radiation and a perspective 
centre associated with the images, the reconstruction processing means being 
arranged to derive said baseline from the correlation. 

3. Apparatus as claimed in claim 1 or claim 2 wherein the image processing means 
is arranged to correlate two or more calibration images and to determine the spacing 
between origins of the first and second projections in dependence upon both the 
correlation of the two or more calibration images and input or stored metric 
information associated with the calibration . 

4. Apparatus as claimed in any preceding claim wherein the reconstruction 
processing means is arranged to vary the spacing between the origins of the first and 
second projections in dependence upon a scaling variable enterable by a user. 

5. Apparatus as claimed in claim 4 including means for displaying the 3D 
representation with a relative scaling dependent upon the value of the scaling 
variable. 

6. Apparatus as claimed in any preceding claim including means for combining two 
or more 3D representations and means for adjusting the relative scaling of the 
representations to enable them to fit each other. 
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7 Apparatus as claimed m any preceding claim wherem the image processing means 
arranged to correlate pixels m one of said images with correspondmg locations in 

the other of said images by comparing the local radiometric distributions associated 
with said pixels and locations respectively. 

8 Apparatus as claimed in claim 7 wherein the image processing means is arranged 
to allow a radiometric and/or geometric distortion during the correlation process. 

9 Apparatus as claimed m any precedmg claim further comprising projector means 
arranged to project the structured optical radiation onto the object surface and at 

least one calibration surface. 

10 Apparatus for deriving a 3D representation of at least part of an object, 
comprising means for projecting sUuctured optical radiation onto the surface of the 
object means for acqumng a 2D image of the projection of the structured optical 
.adiation. and image processing means arranged to derive the 3D representation 
from the distortion of the structure of the projected optical radiation by the object 
surface, the projected structured optical radiation having an irregular radiometric 
and/or colorimetric distribution. 

11 Apparatus as claimed m claim 10 wherein the structured optical radiation 
comprises a distribution of three or more radiometric and/or colorimetric intensity 

values. 

12. Apparatus as claimed in claim 10 or claim 11 wherein the structured optical 
radiation comprises a speckle pattern. 

13 Apparatus as claimed in any of claims 10 to 12 wherem the structured optical 
radiation comprises a fractal pattern whose structure is invariant at different scales. 

14 Apparatus as claimed in any of claims 10 to 13 wherein the projector means is 
arranged to project the structured radiation from a focal point or focal line defined 

by its optics. 

15. Apparatus as claimed in any preceding claim further comprising a camera 
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arranged to acquire the object image and at least one calibration image. 

16. Apparatus as claimed in any preceding claim further comprising at least one 
calibration target which in use is illuminated by the structured radiation. 

17. A method of generating a 3D representation of at least part of an object, wherem 
structured optical radiation is projected onto the surface of the object, a 2D image of 
the projection of the structured optical radiation on the surface is acquired, and the 
3D representation is derived from the distortion of the structure of the projected 
optical radiation by the object surface, the projected sUuctured optical radiation 
having an irregular radiometric and/or colorimetric distribution. 

18. A method of generating a 3D representation of an object from an object image of 
the projection of structured optical radiation onto the object surface and from at least 
one calibration image of the projection of the structured optical radiation onto a 
surface displaced from the object surface, the method comprising the steps of: 

i) correlating at least one calibration image with the object image and optionally 
with a further calibration image; 

ii) simulating a first projection of the object image and a second projection of the 
structured optical radiation, and 

iii) deriving said 3D representation from the mutual intersections of the first and 
second projections. 

19. A method as claimed in claim 18 wherein the first and second projections are 
from a baseline linking an origin of the structured optical radiation and a perspective 
centre associated with the image respectively, said baseline being derived from two 
or more pairs of correlated features. 

20. A method as claimed in claim 23 or claim 24 wherein two or more calibration 
images are correlated and the spacing between origins of the first and second 
projections is determined in dependence upon both the correlation of the two or 
more calibration images and input or stored metric information associated with the 
calibration images. 
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21. A method as claimed m any of claims 18 to 20 wherein the spacing between the 
origins of the first and second projections is varied in dependence upon a scaling 
variable entered by a user 

22. A method as claimed in claim 21 wherein the 3D representation is displayed 
with a relative scaling dependent upon the value of the scaling variable. 

23. A method as claimed in any of claims 18 to 22 wherein two or more 3D 
representations are combined and the relative scaling of the representations is 
adjusted along at least one axis to enable them to fit each other. 

24. A method as claimed in any of claimslS to 23 wherein regions of said images 
are correlated by comparing the local radiometric and/or colorimetric distributions 
associated with said regions. 

25- A method of generating a 3D represenUtion of at least part of an object, wherein 
structured optical radiation is projected onto the surface of the object, a 2D image of 
the projection of the structured optical radiation on the surface is acquired, and the 
3D representation is derived from the distortion of the structure of the projected 
optical radiation by the object surface, the projected structured optical radiation 
having an irregular radiometric and/or colorimetric distribution. 

26. A method as claimed in claim 24 or claim 25 wherein a radiometric and/or 
geometric distortion is allowed between potentially corresponding regions. 

27. Image processing apparatus for deriving a 3D representation of at least part of an 
object from a 2D image of the illuminated object, the object being illuminated with 
structured optical radiation projected from a location spaced apart from the 
viewpoint at which the 2D image is acquired, the 2D image being correlated with 
the structured radiation, the apparatus comprising digital processing means arranged 
to form a 3D reconstruction which extends in a simulated 3D space in dependence 
upon both the correlation and a scaling variable, the scaling variable being 
representative of the separation between the location from which the structured 
optical radiation is projected and the viewpoint at which the 2D image is acquired. 
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28 Apparatus as claimed in claim 27 which is arranged to derive a further 3D 
reconsuuction from a further 2D image acquired from a different viewpoint relative 
to the object, the combining means being arranged to combine the first-mentioned 
3D reconstruction and the further 3D reconstruction by manipulations in a simulated 
3D space involving one or more of rotation and translation, the apparatus further 
comprising scalmg means arranged to reduce or eliminate any remainmg 

discrepancies between the 3D reconstructions by scaling one 3D reconstruction 

relative to the other along at least one axis. 

29 Apparatus as claimed in claim 28 which is arranged to display both 3D 
reconstructions simuluneously and to manipulate them in simulated 3D space in 
response to commands entered by a user. 

30 Apparatus as claimed m claim 29 which is arranged to perform the 
manipulations of the 3D reconstructions under the control of a computer pointing 

device. 

31 A method of deriving a 3D representation of at least part of an object from a 2D 
image thereof, comprising the steps of illuminating the object with structured 
projected optical radiation, acquiring a 2D image of the lUurmnated object, 
correlating the 2D image with rays of the structured optical radiation, and digitally 
processing the 2D image to form a 3D reconstruction which extends in a simulated 
3D space in dependence upon both the correlation and a scaling variable, the scaling 
variable being representative of the separation between a location from which the 
structured optical radiation is projected and the viewpoint at which the 2D image is 
acquired. 

32. A method as claimed m claim 31 wherem a view of the reconstruction in the 
simulated 3D space is displayed on a screen and the scaling variable is entered by a 



user. 



33. Apparatus for generating a 3D representation of at least part of an object, 
substantially as described hereinabove with reference to Figures 1 to 4 and 11 
optionally modified in accordance with Figures 5 to 10 of the accompanying 

drawings. 
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34. A method of deriving a 3D representation of at least part of an object, 
substantially as described hereinabove with reference to Figures 1 to 4 and 11 
optionally modified in accordance with Figures 5 to 10 of the accompanying 
drawings. 
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Abstract 



An apparatus for deriving a 3D representation of at least part of an object (3) 
comprises a projector (PR) arranged to project a speckle pattern or other structured 
light from a slide (S) onto the object and a single digital camera (CM) arranged to 
acquire an image of the object and an image of a cahbration target (T) which is 
similarly illuminated by the speckle pattern in a preliminary calibration step. Pairs of 
points (PI, Ql; P2. Q2) of the respective images are correlated eg by Gnien's 
algorithm to enable the baseline vector (V) joining the perspective centres (Oo Op) 
of the camera and projector to be found from the intersection of planes OcPlQl 
and OcP2Q2. The 3D representation is then generated in simulated 3D space by 
projection and intersection of ray bundles from virtual projectors (implemented in 
software) located on the baseline vector (V). 
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